Establishing the Upper Bound and Inter-judge Agreement of a Verb Classification Task
نویسندگان
چکیده
Detailed knowledge about verbs is critical in many NLP and IR tasks, yet manual determination of such knowledge for large numbers of verbs is difficult, time-consuming and resource intensive. Recent responses to this problem have attempted to classify verbs automatically, as a first step to automatically build lexical resources. In order to estimate the upper bound of a verb classification task, which appears to be difficult and subject to variability among experts, we investigated the performance of human experts in controlled classification experiments. We report here the results of two experiments—using a forced-choice task and a non-forced choice task—which measure human expert accuracy (compared to a gold standard) in classifying verbs into three pre-defined classes, as well as inter-expert agreement. To preview, we find that the highest expert accuracy is 86.5% agreement with the gold standard, and that inter-expert agreement is not very high (K between .53 and .66). The two experiments show comparable results.
منابع مشابه
Task-Specific Artifacts of Parametric Properties in English as a Second Language Acquired by Persian-Speaking Learners
This experimental study investigated the learners’ integrative acquisition of obligatory overt subjects and subject-verb clause agreement in English as an L2. In L1 acquisition research, correlations between superficially unrelated linguistic phenomena are analyzed in terms of integrative effects. For instance, in English L1 acquisition, there is evidence for an integrative appearance of subjec...
متن کاملA generalized upper bound solution for bimetallic rod extrusion through arbitrarily curved dies
In this paper, an upper bound approach is used to analyze the extrusion process of bimetallic rods through arbitrarily curved dies. Based on a spherical velocity field, internal, shearing and frictional power terms are calculated. The developed upper bound solution is used for calculating the extrusion force for two types of die shapes: a conical die as a linear die profile and a streamlined di...
متن کاملAn Upper Bound Approach for Analysis of Hydroforming of Sheet Metals
Considering a kinematical velocity admissible field, the upper bound method has beenused for predicting the amount of pressure in hydroforming of sheet metals. The effects of workhardening, friction and blank size have been considered in pressure prediction. Also the effect ofsheet thickness variation has been considered in the present work formulations. The relation betweenpressure and punch s...
متن کاملA Study of Precision Forging of Steel Spur Gears: Upper Bound Model and Experiments
Precision forging process is widely used for the production of spur gears due to its advantages such as improved strength and surface finish, little waste of materials and reduction in the machining time of the gear. An important aspect of precision forging process is the load required to perform the process successfully and design the forging tools. In this research, a new kinematically admiss...
متن کاملTask-space Control of Electrically Driven Robots
Actuators of robot operate in the joint-space while the end-effect or of robot is controlled in the task-space. Therefore, designing a control system for a robotic system in the task-space requires the jacobian matrix information for transforming joint-space to task-space, which suffers from uncertainties. This paper deals with the robust task-space control of electrically driven robot manipula...
متن کامل